Truth and Bias, Left and Right: Testing Ideological Asymmetries with a Realistic News Supply

Abstract The debate around “fake news” has raised the question of whether liberals and conservatives differ, first, in their ability to discern true from false information, and second, in their tendency to give more credit to information that is ideologically congruent. Typical designs to measure these asymmetries select, often arbitrarily, a small set of news items as experimental stimuli without clear reference to a “population of information.” This pre-registered study takes an alternative approach by, first, conceptualizing estimands in relation to all political news. Second, to represent this target population, it uses a set of 80 randomly sampled items from a large collection of articles from Google News and three fact-checking sites. In a subsequent survey, a quota sample of US participants (n = 1,393) indicate whether they believe the news items to be true. Conservatives are less truth-discerning than liberals, but also less affected by the congruence of news.

Supplementary Materials file is based on a fully reproducible RMarkdown file that contains code to replicate all graphs and analyses, which can be found at https://osf.io/82w7u/.

A Pennycook & Rand (2019) replication
The authors of the original study did not only ensure that there was an equal number of pro-Clinton and pro-Trump items. They also balanced on the continuum of ideological valence, so that pro-Clinton items were pretested as equally distant from the ideology midpoint as pro-Trump items (as communicated in an email exchange). To account for this, I filter out sub-samples that do not meet this criterion in the below plot. Of my 500 sub-samples, 0.94 are balanced this way. Of those, only 0.53 are significant, supporting the point made in main paper.

Data collection from Google News
The Google News data was collected through the Google News API with the following parameters: "from" and "to" in order to define the time frame; "language" set to "en" to limit results to English news items; "sources" to a list of sources determined through the endpoint https://newsapi.org/v2/sources with parameter "country" set to "us" and "category" to "business", "general", "health" or "science" to exclude news that are likely non-political.

B.2 Data collection from fact-checkers
A first risk in relying on fact-checkers is a potential ideological bias in selection what information to check (Ostermeier 2011; Uscinski and Butler 2013; Marietta, Barker, and Bowser 2015). Second, some fact-checker only publish about news checked as false. The aim of publishing only the latter might distort the selection process (Marietta 2019). A third risk is that although fact-checkers have been found to be reliable on outright truths and falsehoods, ratings of ambiguous claims are more unreliable (Nieminen and Rapeli 2019; Lim 2018). To minimize these risks, I excluded fact-checkers that only publish false results. I further excluded fact-checkers that primarily check speeches by politicians. I ended up with the three organizations Snopes, Politifact and Truth or Fiction.
The risk of selection bias cannot be excluded completely. At least in theory, all three organisations commit to non-partisan selection, although procedures do not seem formalized. Snopes explains that they "don't choose or exclude items for coverage based on whether they deal with Republican/Democratic, conservative/liberal, or religious/secular issues" and that "reader interest" plays a role. 1 PolitiFact lists several selection criteria, among them whether a statement is "significant" and commits "to check from both Democrats and Republicans". 2 Truth or Fiction describes their focus as "on stories that are the most widely-circulated via social media". 3 The fact-check collection was composed through scraping the archives of Snopes, Politifact and Truth or Fiction for the same time frame with the following parameters: • https://www.snopes.com/fact-check/category/politics/; reports labelled as "mixture", "unproven", "outdated", "miscaptioned", "scam", "legend", "labeled satire", "lost legend", "mostly true" and "mostly false" were not included; those labelled as "false", "misattribution", "true" or "correct attribution" were included. • https://www.politifact.com/factchecks/; categories "Facebook Post", "Bloggers" and "Instagram" included; "Viral Image" and "Names" exluded; reports labelled as "True", "False" or "Pants on Fire" included, "Mostly True", "Half True" and "Mostly False" excluded.
Below, Section B.3 list the items excluded during the iterative sampling procedure, separately for the two collections. Exclusion reasons as explained in the paper. Section B.4 lists the URLs of the items in the final sample. Each item has a unique ID that corresponds to the ID in Table 5.

B.5 Ideological valence pre-testing
Ideological valence of the 80 news items was pre-tested between June 19 and 30, 2020 with two different opt-in samples on Prolific and MTurk. To improve quality of raters, MTurk participants had to have voted in the 2016 presidential elections. Prolific raters also had to have indicated "politics" has a hobby, and were recruited so that there was an equal number of moderates, liberals and conservatives.
After giving consent and answering some sociodemographic questions, respondents were introduced to the task with the following text: "We would like you to look at about ten news reports/posts we have found online. For each of them, we would like you to assume that the information is completely accurate, and to answer the following questions: 1) Is this information more favorable to conservatives or liberals or neither group? 2) Is this information more consistent with the beliefs of conservatives, or of liberals, or of neither group? The answer options go from '-2 (liberals)' to '2 (conservatives)'. A good part of the reports/posts might not have such relevance for either side. In that case, do not hesitate to choose '0'!" In the main task, each respondent saw a random subset of 10 out of the 80 news items. For each item, respondents were asked to respond to two questions: "Assume the above information is entirely accurate. Is it-or was it at the time of publishing-more favorable to liberals or conservatives or neither?" and "Assume the above information is entirely accurate. Is it-or was it at the time of publishing-more consistent with the beliefs of liberals or of conservatives or neither?", both with response scales from "-2 (liberals)" to "2 (conservatives)". After rating the ten items, some more covariates were measured and respondents were debriefed, i.e., received corrections about any false items they saw.
Figure 2 below shows that item-level average ratings are highly correlated both between measures and between samples. Table 5 lists all items along with their truth value, and the average valence rating (number of raters per item and standard error in parenthesis).      1.24 (n=51, SE=0.14)

C Subject sample
Tables 6 through 9 compares the sample with population statistics on basic sociodemographic variables.    "What is your gender?" "Male", "Female", Other" Education "What is the highest level of school you have completed or the highest degree you have received?" 5 choices from "12th grade without diploma or less" to "Master's degree, professional school degree, or doctorate degree" Partisanship "Generally speaking, do you usually think of yourself as a Democrat, a Republican, an Independent, or what?" "Democrat", "Republican", "Independent", "Other party", "No preference" Ideology "Here is a seven-point scale on which the political views that people might hold are arranged from extremely liberal to extremely conservative. Where would you place yourself on this scale? 7-point scale from "Extremely liberal" to "Extremely conservative" and "Haven't thought much about this" Turnout "In 2016 Hillary Clinton ran on the Democratic ticket against Donald Trump who ran for the Republicans. Do you remember for sure whether or not you voted in that election?" "Yes, voted", "No, didn't vote" Facebook use "How often do you use Facebook?" 9-point scale from "Many times every day" to "Never" and "I don't have an account" General media trust "In general, how much trust and confidence do you have in the mass media -such as newspapers, TV and radio -when it comes to reporting the news fully, accurately, and fairly?" 7-point scale from "0none at all" to "6 -a great deal" Digital literacy "How familiar are you with the following computer and Internet-related items? Please choose a number between 0 and 6 where 0 represents 'no understanding' and 5 represents 'full understanding' of the item." Separate 7-point scales for "Pishing", "Hashtag", "JPG", "Malware", "Cache", "RSS" 27 Variable Question Choices/Coding Economic left-right item 1 "Some people think the government should provide fewer services even in areas such as health and education in order to reduce spending, others feel it is important for the government to provide many more services even if it means an increase in spending.
Where would you place yourself on a scale from 1 to 7?" 7-point scale from "0 -Government should provide fewer services" to "6 -Government should provide more services" Economic left-right item 2 "Some people feel the government in Washington should see to it that every person has a job and a good standard of living. Others think the government should just let each person get ahead on their own. Where would you place yourself on a scale from 1 to 7?" 7-point scale from "0 -Government should let people get ahead on their own" to "6 -Government should see to jobs and living standards" Social left-right item 2 "There has been some discussion about abortion during recent years. Which one of the opinions below best agrees with your view?" 4 choices from "By law, abortion should never be permitted" to "By law, a woman should always be able to obtain an abortion as a matter of personal choice." Social left-right item 2 Do you think gay or lesbian couples should be legally permitted to adopt children?
"Yes", "No" Social left-right item 3 "Do you think it is better, worse, or makes no difference for the family as a whole if the man works outside the home and the woman takes care of the home and family?" "Worse", "Better", "Makes no difference" Need for closure item 1 "How disorganized are the rooms that you personally live and work in most?" 7-point scale from "0not disorganized at all" to "7 -Extremely disorganized" Need for closure item 2 "Do you like unpredictable situations or dislike them?" 7-point scale from "0dislike unpredictable situations" to "7extremely disorganized" Need for closure item 3 "How many of your important decisions do you make quickly and confidently?" 7-point scale from "0 -None" to "6 -All" Need for closure item 4 "When you don't understand the reason why something happens in your life, how uncomfortable does that make you feel?" 7-point scale from "0 -Not uncomfortable at all" to "6 -Very uncomfortable" 28 Variable Question Choices/Coding Need for closure item 5 "In the situations when you see two people in a conflict with one another, how often can you see how both sides could be right?" 7-point scale from "0 -Always" to "6 -Never" Cognitive reflection item 1 "A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?" Open-ended Cognitive reflection item 2 "In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake?" Open-ended

F.2 Regressions with covariates
The regression below include interactions with some of the variables shown in Figure 3 above. The left columns of tables 11 and 12 shows the same models as reported in the main text. The middle columns include interactions of the truth or congruence of the news item with all variables correlated to ideology. The rightmost column includes only variables with a significant interaction term, and serves as the basis for the marginal-effects plots. In sum, it shows that none of the measured variables can explain away the ideological asymmetries found. But some variables reveal interesting interactions in themselves: For example, higher age, higher digital literacy and greater media trust all contribute positively to truth discernment, as seen in Figure 4.      Figure 7: Effect of controlling for valence delta on truth discernment asymmetry As shown in Figure 2 in the main paper, false items have a valence score that is slightly higher, i.e. more conservative, than true items. I refer to this difference as "valence delta". To understand how results depend more generally on the valence delta, I proceed as follows: Out of the complete sample of 80 items, I sample 20 true and 20 false items 50 times. To vary the extent to which false and true items differ in ideological valence across samples, I set different constraints while sampling: 10 samples are constrained to have a valence delta smaller or equal to zero; 10 samples a delta smaller than 0.1; further 10 a delta smaller than 0.3; further 10 a delta smaller than 0.5; and 10 are sampled with no constraint. For each of these samples, I run the same cluster-robust regression and record the relevant coefficients each time. Figure 7 shows the truth-ideology interaction coefficient from all these regressions, plotted against the valence delta. Significant cases are colored in dark. As previously, a negative interaction coefficient means that conservatives are less truth discerning. It can be seen that with a small valence delta, the ideological asymmetry indeed decreases, which suggests that supply does matter. Overall, the correlation between valence delta and the regression coefficient is -0.56. However, among the samples with a valence delta around zero, a good part still reveals an asymmetry. These alternative estimates with a valence data of zero are more comparable with studies like Pennycook and Rand (2019). On the one hand, given that imperfectness of my data collection, there may be value in such balancing. On the other hand, it is still unclear what kind of population of information such balanced sub-samples represent. What is more, there are as many ways to conduct such balancing as there are many item-level variables potentially relevant. For example, familiarity with information has been shown to strongly influence belief (Brashier and Marsh 2020). To explore how balancing the sample affects asymmetries in bias, I apply the sample resampling strategy as in the previous subsection. For each of these samples, I run the same cluster-robust regression and record the relevant coefficients each time. Figure 8 shows the truth-ideology interaction coefficient from all these regressions, plotted against the valence delta. Significant cases are colored in dark. A negative interaction coefficient means that conservatives are less biased. It can be seen that with a small valence delta, the ideological asymmetry indeed decreases,. Overall, the correlation between valence delta and the regression coefficient is -0.79. Table 13 shows results for RQ2 and RQ3 using a continuous version of the congruence variable, instead of the categorical version.  Table 14 show results for RQ2 and RQ3 using a the "favorability" instead of "consistence" measure of ideological valuence

I.3 Including subjects who did not finish
Tables 15 shows results when those who did not finish the questionnaire are included, which gives a sample size of n = 1641.  Table 16 shows results for RQ1, RQ2 and RQ3 using partisanship instead of ideology as the underlying political predisposition.